Skip to content

feat(webapp,run-engine,core,clickhouse): surface combined concurrency in metrics and dashboard - #4830

Open
matt-aitken wants to merge 19 commits into
feat/queue-concurrency-overridesfrom
feat/queue-concurrency-metrics
Open

feat(webapp,run-engine,core,clickhouse): surface combined concurrency in metrics and dashboard#4830
matt-aitken wants to merge 19 commits into
feat/queue-concurrency-overridesfrom
feat/queue-concurrency-metrics

Conversation

@matt-aitken

@matt-aitken matt-aitken commented Aug 29, 2026

Copy link
Copy Markdown
Member

Summary

Queues with a combinedConcurrencyLimit now show how they use it, in the dashboard and the API. Stacked on #4829.

The queues list gains a Total column showing runs in flight across all concurrency keys against the queue's total cap, turning warning-colored when the cap is reached. The queue detail page gains a "Total concurrency" chart plotting total running against the effective cap over time. Queue retrieve and list API responses include the same totals under concurrency.combined.

Design

The queue metrics gauge (emitted from the dequeue and enqueue Lua scripts) is extended with two fields: total running (the group concurrency set's cardinality) and the raw stored total limit. They flow through the existing metrics stream into the ClickHouse raw table and are aggregated as max_total_running / max_total_limit in the 10s and 5m rollup tiers, queryable from the dashboard's TRQL widgets. The per-key rollup additionally records the queue limit that applied to each key.

Reading the live numbers is O(pages shown): the list presenter batches one SCARD per queue that has a cap.

@changeset-bot

changeset-bot Bot commented Aug 29, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 6d809e1

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0519a303-e367-4f9c-ad82-85fe8a7de85c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Devin Review

@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch from 8f5ad5d to c60df90 Compare August 29, 2026 14:49
@pkg-pr-new

pkg-pr-new Bot commented Aug 29, 2026

Copy link
Copy Markdown

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@6d809e1

trigger.dev

npm i https://pkg.pr.new/trigger.dev@6d809e1

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@6d809e1

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@6d809e1

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@6d809e1

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@6d809e1

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@6d809e1

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@6d809e1

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@6d809e1

commit: 6d809e1

@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch from c60df90 to 2816d30 Compare August 29, 2026 15:33

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Devin Review

@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch from fe6772f to 6bc8bef Compare August 29, 2026 15:52

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Devin Review

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Devin Review

devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Devin Review

@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch from e802dab to 1627de4 Compare August 29, 2026 18:11
@matt-aitken matt-aitken changed the title feat(webapp,run-engine,core,clickhouse): surface total concurrency in metrics and dashboard feat(webapp,run-engine,core,clickhouse): surface combined concurrency in metrics and dashboard Aug 29, 2026
devin-ai-integration[bot]

This comment was marked as resolved.

@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch from 1627de4 to f4b8109 Compare August 29, 2026 18:18
@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch from f4b8109 to ecfbdbb Compare August 29, 2026 18:21
devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Devin Review

devin-ai-integration[bot]

This comment was marked as resolved.

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Observability map

As of 6d809e1.

Nothing in this pull request moves the report any more. The findings an earlier push reported are gone.

The score and findings here are report-only and never gate the merge. Separately, a required test suite keeps this tool's symbol and route lists in sync with the code they name, and can fail a pull request that renames or removes a symbol they reference, or that adds the first route with a segment they anticipate. Each failure names the list to edit. The rules and their reasons: internal-packages/observability-map/README.md.

devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Devin Review

@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch 3 times, most recently from c51acdd to aba9d48 Compare August 31, 2026 12:33
@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch from aba9d48 to b6593d5 Compare August 31, 2026 12:38
@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch from b6593d5 to 40a8ecc Compare August 31, 2026 12:38

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Devin Review

… metrics and dashboard

Queues with a totalConcurrencyLimit now report how they use it. The gauge
pipeline emits total running and the stored cap, ClickHouse aggregates them
into the queue metrics tiers, the queues list gets a Total column, the queue
detail page charts total running against the cap, and the per-key table shows
each key's effective limit including per-key overrides. Queue retrieve and
list API responses include the same totals.
…tore views on rollback

The CK enqueue gauges (fast path and queued path) now sample total running
and the stored cap, so metric buckets fed only by enqueues no longer record
zero totals. The migration's down section recreates the pre-existing
materialized view definitions so ingestion keeps flowing after a rollback.
The per-key table reads only the page's overrides with one HMGET instead of
loading the queue's whole override hash.
The test harness splits a migration's up section on semicolons, so a
semicolon inside a comment yields a comment-only statement that ClickHouse
rejects as an empty query and every container-backed suite fails at setup.
Queue retrieve only asks the engine for total running when a total limit is
set, matching the list presenter and avoiding a pointless read for the
common uncapped case.
The total concurrency numbers come from live Redis, not the metrics
pipeline, so the column belongs in both tables rather than only behind the
queue metrics UI gate.
A separate Total column implied every queue should have one, and its dash
read as a missing limit on queues that never use concurrency keys. Only
queues that declare a totalConcurrencyLimit now change: their Limit cell
reads as per-key plus total (e.g. 1 /key, 3 total) and Running turns
warning-colored when the total cap is saturated. Plain queues are
unchanged.
The total cap gates keyed admissions, so the Running cell now warns off the
group count rather than the aggregate that also includes unkeyed runs.
…dashboard and metrics

Queue API responses expose concurrency.combined, the dashboard says
combined, and the new metrics columns are named combined_running and
combined_limit.
Queues that set a combinedConcurrencyLimit show it bracketed next to the
per-key limit with a fine dashed underline and an explanatory tooltip; the
Limit header tooltip is width-capped. Queues without one are unchanged.
The tooltip trigger is a button, so nesting it in the Limit cell's link made
clicking it navigate; it now renders as the cell's trailing adornment.
Also reflows an import to the formatter's current output.
The dequeue gauge ran before the loop, so a queue's first batch from idle
and its final drain were never sampled with their runs in flight and the
combined chart under-reported. The successful path now re-samples after
admissions; early returns keep the entry sample. Also documents that
combined.current is the declared cap, clamped at admit time.
…mit column

Removes the override-aware admit and gauge reads from the CK Lua scripts and
the per-key limit column, following the removal of runtime per-key overrides
from this stack.
The class the ignore covered is deleted at this level, so the merged result
carries no dead-code exemption.
The plain dequeue gauge re-samples after admissions like the keyed one, the
repair path clears a keyed run's variant and group slots by concurrency key,
pause responses include the combined limit, stale wording and a leftover
changeset from before the rename are cleaned up, and two empty flag blocks
are removed from the fast-path scripts.
@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch from e49c3f5 to 6d809e1 Compare August 31, 2026 13:04

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Devin Review

Comment on lines +95 to +97
totalConcurrencyLimit: updatedQueue.totalConcurrencyLimit ?? null,
totalConcurrencyLimitBase: updatedQueue.totalConcurrencyLimitBase ?? null,
totalConcurrencyLimitOverriddenAt: updatedQueue.totalConcurrencyLimitOverriddenAt ?? null,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Pause responses lose combined usage

When a capped queue is paused or resumed, toQueueItem receives no combined running count. The API returns concurrency.combined.running as null despite active keyed runs.

Prompt for agents
Update PauseQueueService.call in apps/webapp/app/v3/services/pauseQueue.server.ts to read the queue's combined running count when totalConcurrencyLimit is present, following QueueRetrievePresenter's conditional engine.totalConcurrencyOfQueues call. Pass that value as totalRunning to toQueueItem so the pause/resume API response reports the same live combined usage as retrieve and list responses without adding an unnecessary Redis read for uncapped queues.
Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant